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The  Independent  Readiness  Review  Team  (IRRT)  performs  risk  assessment  of  space 
launch  missions  and  reports  findings  in  pre-launch  reviews  to  SMC  Leadership.  To 
date,  the  IRRT  has  performed  reviews  on  the  Evolved  Expendable  Launch  Vehicle 
(EELV)  Atlas  V  and  Delta  IV  boosters,  Heritage  boosters  such  as  Titan  II,  Delta  II 
and  Minotaur,  and  payloads  including  DSP,  GPS,  SBIRS,  DMSP,  Milstar,  AEHF, 
WGS  and  selected  space  test  programs.  Since  the  inception  of  the  IRRT,  SMC  has 
experienced  an  unprecedented  46  straight  mission  successes  in  a  row. 


I.  Introduction 

The  history  of  Mission  Assurance  at  Space  and  Missile  Command  (SMC)  is  that 
independent  review  teams  were  formed  close  to  mission  launch  to  perform  reviews  of 
large  launch  vehicles.  These  teams,  which  consisted  of  Air  Force,  Aerospace  and 
contractors  would  review  the  mission,  present  their  risk  assessment  and  then  be 
disbanded.  During  Acquisition  Reform  of  the  mid  1990s,  SMC  eliminated  Independent 
Reviews  although  NRO  retained  their  Mission  Assurance  Team  (MAT)  for  launch 
vehicles. 

In  the  late  1990’s,  the  Air  Force  experienced  repeated  failures  of  Delta  II  and 
Titan  IV  systems.  These  failures  were  attributed  to  the  lack  of  discipline  that  resulted 
from  Acquisition  Reform.  In  2001,  Lt.  General  B.  Arnold  SMC/CC  reformed  and 
reconstituted  the  independent  review  process  for  SMC  missions  in  response  to  the 
succession  of  Broad  Area  Reviews  (BAR)  conducted  subsequent  to  the  failures. 

The  primary  change  from  previous  independent  reviews  was  the  formation  of  a 
standing  team,  The  Independent  Readiness  Review  Team  (IRRT),  to  perform  readiness 
reviews  of  both  space  launch  vehicles  and  payloads.  The  IRRT  was  made  independent  of 
all  two-letter  organizations  and  reported  directly  to  the  SMC  commander.  The  IRRT  was 
given  its  own  annual  budget  funded  by  the  System  Program  Offices  through  the  above 
standard  cost  mechanism. 
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The  IRRT  charter  is  to  perform  independent  reviews  of  Air  Force  satellite 
acquisition  and  launch  missions  as  directed  by  SMC/CC.  The  reviews  are  performed  to 
support  the  SMC  Space  Flightworthiness  Certification  process  as  dictated  by  SMCI  63- 
1201,  Assurance  of  Operational  Safety,  Suitability  &  Effectiveness  (OSS&E)  for  Space 
and  Missile  Systems,  and  defined  in  SMCI  63-1202,  Space  Flight  Worthiness.  Other 
documents  describing  the  review  process  are  SMCI  63-1203,  Independent  Readiness 
Reviews,  and  SMCI  63-1204,  Readiness  Review  Process.  The  objective  of  the  reviews  is 
to  identify  technical  risks,  make  recommendations  for  mitigation  and  to  provide 
independent  assessment  of  launch  readiness.  The  IRRT  formally  presents  the  results  at 
periodic  reviews  and  reports  directly  to  SMC  Leadership. 

To  fulfill  its  charter,  the  IRRT  follows  a  number  of  guiding  principles.  The  IRRT 
strives  always  to  be  value  added  to  the  SPO,  SMC/CC  and  the  Chief  Engineer.  This  is 
accomplished  in  a  number  of  ways  including  1)  a  proactive  and  phased  review  that 
leverages  existing  SPO  processes,  and,  2)  the  maintenance  of  strong  communication  links 
including  regular  status  briefs  to  ensure  early  identification  of  issues  and  concerns.  The 
benefit  of  this  approach  is  that  it  raises  IRRT  identified  issues  early  in  the  process  while 
adequate  time  exists  prior  to  mission  launch  to  allow  recommendations  to  be 
implemented.  Cost  is  also  minimized  since  the  majority  of  IRRT  reviews  are  inline  with 
the  SPO,  thus  eliminating  the  need  for  costly  separate  reviews. 


II.  IRRT  Organization 

The  IRRT  consists  of  Air  Force,  Aerospace  and  SETA  contractors  as  shown  in 
Figure  1  below.  The  IRRT  is  billeted  for  one  Lt.  Col  who  acts  as  the  Chief,  four  full-time 
Majors  and  two  Lieutenants.  The  Aerospace  Corporation  has  ten  full-time  employees  in 
the  IRRT  program  office  including  the  Vice-Chief.  Additionally,  Aerospace  has  support, 
on  an  as-needed  basis,  from  their  Engineering  and  Technology  Group  (ETG).  Finally,  the 
IRRT  receives  substantial  support  from  three  SETA  contractors,  Northrop  Grumman 
(NGC),  SAIC,  and  ARINC. 

The  IRRT  is  a  matrix  organization  with  both  System  and  Panel  Leads  that  are 
responsible  for  accomplishing  the  required  reviews.  The  system  leads  are  responsible  for 
the  review  of  a  specific  system  and  are  given  budget  authority  and  resources  to 
accomplish  the  required  tasks.  The  panel  leads  support  multiple  system  leads  in  specific 
disciplines  such  as  propulsion,  avionics,  software,  mechanical/structural,  etc.  To 
illustrate  how  the  IRRT  functions,  consider  a  typical  review  of  the  Delta  II  launch 
vehicle.  The  system  lead  responsible  for  Delta  II  will  request  support  from  each  of  the 
panels  to  perform  reviews,  identify  any  issues  and  assist  in  preparation  of  risk  charts.  The 
panels  that  support  the  Heritage  System  Lead  include  propulsion,  structures,  avionics, 
software  and  product  assurance.  Each  panel  lead  will  request  support  in  performing  the 
required  reviews  from  the  appropriate  engineers  supporting  the  specific  panel.  The 
propulsion  panel  reviews  will  include  the  Graphite  Epoxy  Motors  (GEM),  which  are  the 
solid  strap-on  motors  and  the  stage  1  RS-27  and  stage  2  AJ-10  liquid  Engine  Pedigree 
Reviews.  Booster  integration  reviews  may  also  be  supported  by  propulsion.  The  results 
from  the  reviews  performed  by  each  panel  are  forwarded  to  the  system  lead.  He  converts 
them  into  risk  charts.  These  charts  are  reviewed  and  approved  by  the  IRRT  Chair  and 
Vice-Chair  prior  to  release  to  the  Delta  SPO  and  SMC  leadership. 
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Figure  1.  The  standing  team  of  the  SMC  IRRT  is  composed  of  Air  Force,  Aerospace, 
Northrop-Grumman,  SAIC  and  ARINC  engineers. 


III.  IRRT  Review  Approach 

The  IRRT  employs  a  team  approach  to  all  review  activities  that  includes  1) 
Review  of  SPO  and  contractor  data  focusing  on  the  high  risk  areas,  2)  Participation  in 
program  processes,  3)  Utilization  of  a  clearly  defined  issue  identification  and  resolution 
process,  and,  4)  Closure  and  risk  assessment. 

The  IRRT  review  of  SPO  and  contractor  documentation  focuses  on  a  number  of 
items  as  discussed  below  that  are  believed  to  pose  the  greatest  risk  to  the  success  of  space 
launch  missions.  By  focusing  efforts  on  the  high-risk  items,  the  IRRT  can  identify  issues 
of  concern  in  a  cost-effective  manner. 

1.  Test  as  you  fly  exceptions 

One  of  the  most  important  lessons  learned  in  the  space  launch  business  is  the 
necessity  to  perform  tests  or  analysis  of  hardware  and  software  in  the  same 
manner  the  mission  is  flown.  Exceptions  to  test  as  you  fly  have  resulted  in  past 
flight  failures  and  represents  an  increased  program  risk  that  must  be  appropriately 
addressed. 

2.  Critical  Qualification  Margins 

Qualification  margins  of  critical  items  are  of  high  importance.  Hardware  that  has 
minimal  margins  of  safety  poses  an  increased  risk  to  failure  due  to  variations  in 
mechanical  properties,  performance  or  other  critical  measures.  Hardware  with 
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large  margins  has  rarely  been  associated  with  mission  failure,  thus  requiring  less 
review. 

3.  First  flight  items 

First  flight  items  receive  increased  scrutiny  simply  because  they  have  not  been 
demonstrated  to  work  under  actual  flight  conditions.  The  IRRT  requests  a  list  of 
first  flight  items  at  each  review  along  with  a  clear  description  of  the  qualification 
performed  to  ensure  the  first  flight  item  will  perform  adequately. 

4.  Single  point  failures 

Redundancy  in  a  system  is  desired  since  it  significantly  reduces  the  probability  of 
failure.  Potential  single  point  failures  must  be  carefully  scrutinized  to  ensure  an 
adequate  level  of  quality. 

5.  Non-conformances 

Non-conformances  occur  when  hardware  or  software  does  not  meet  the 
specification  requirements.  In  some  instances,  the  item  will  be  reworked  to 
correct  the  discrepancy  while  in  other  instances  it  will  be  recommended,  “use  as- 
is”.  A  review  of  the  specific  actions  performed  and  the  contractor  justification  is 
considered  a  high  priority.  The  IRRT  has  identified  instances  where  contractor 
justification  to  Use-as-is  was  inadequate  resulting  in  replacement  of  questionable 
hardware. 

6.  Anomalies 

Anomalies  represent  situations  where  hardware  or  software  did  not  perform  as 
expected.  Careful  review  and  test  or  analysis  is  required  to  verify  the  anomaly 
will  not  occur  or  will  not  have  a  significant  impact  on  the  mission  for  which  the 
reviews  are  performed. 

7.  Escapements 

Escapements  represent  an  event  wherein  the  contractor  missed  something  such  as 
releasing  hardware  that  did  not  receive  all  of  the  required  testing.  The  IRRT 
strives  to  identify  any  escapements  as  part  of  the  pedigree  review  and  hardware 
acceptance  review  process.  Once  identified,  escapements  are  reviewed  carefully 
to  assess  the  likely  impact  on  mission  performance  and  recommendations  are 
made  for  corrective  action.  This  may  include  test  or  analysis  or  simply  use  as-is. 

8.  Unverified  Failures 

Unverified  failures  are  instances  when  failures  occur  and  root  cause  is  not 
identified.  Unverified  failures  are  of  concern  since  confidence  in  hardware  or 
software  fixes  is  low  if  the  root  cause  of  the  problem  cannot  be  identified.  These 
failures  are  of  most  concern  when  there  is  potential  for  mission  failure. 

9.  Out-of-position/  Sequence  (OOPS)  work 

Occasionally,  contractors  will  deviate  from  their  paperwork  and  perform  work  out 
of  sequence.  This  may  result  in  errors  in  the  product  that  need  to  be  addressed  or 
test  results  that  are  invalid.  The  IRRT  reviews  such  errors  to  evaluate  impact  on 
the  mission  and  provides  recommendations  for  corrective  action. 

10.  Out-of- family  results 

Out-of-family  results  are  carelully  reviewed  since  they  often  indicate  that 
something  has  changed  in  the  process.  Statistical  Process  Control  (SPC)  is  an 
effective  means  to  identify  out-of-family  results. 
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The  IRRT  is  unique  for  a  review  team  in  that  it  participates  in  the  program 
processes.  The  IRRT  supports  1)  Technical  Interchange  Meetings,  2)  Integrated  Product 
Team  Meetings,  3)  Hardware  Acceptance  Reviews  (HARs),  4)  Pedigree  Reviews,  and  5) 
Design  Reviews.  The  IRRT  follows  the  SPO  and  contractor  processes  and  provides 
opinions  and  recommendations  for  consideration.  We  support  problem  resolution  and 
failure  investigation  activities  as  appropriate  such  as  nozzle  delamination  investigations 
and  SRM  redesign  activities.  The  benefit  is  the  IRRT  has  both  input  and  insight  into  the 
SPO  and  contractor  actions  as  opposed  to  teams  that  perform  reviews  after  the  fact. 
Despite  its  interaction  with  the  SPOs  and  the  contractors,  the  IRRT  remains  independent 
with  careful  adherence  to  its  processes  and  procedures. 

The  IRRT  follows  a  clearly  defined  issue  identification  and  resolution  process. 
This  includes  inline  process  participation  as  noted  previously  and  formal  submission  of 
requests  for  information,  RFIs,  and  concerns.  The  IRRT  maintains  an  extensive  database 
documenting  all  identified  issues  and  resolution  to  allow  virtual  collaboration  and  to 
retain  lessons  learned.  The  use  of  SETAs  has  also  proven  useful  since  these  companies 
often  bring  knowledge  and  experience  to  bear  in  problem  resolution  that  otherwise  would 
not  be  available.  For  instance,  Northrop  Grumman  has  over  40  years  of  ICBM  experience 
that  has  been  useful  in  the  resolution  of  problems  with  SRM  nozzle  defects  and  the  SRM 
design  issues.  The  IRRT  maintains  close  communication  through  weekly  status  reports 
that  identify  current  risk  ratings,  issues  and  scheduled  meetings.  Closure  of  RFIs  is 
performed  in  a  formalized  manner  by  the  team  leadership  and  typically  involves  internal 
technical  meetings. 


IV.  IRRT  Deliverables 

The  IRRT  examines  selected  parts,  components,  subsystems  and  subsystem 
pedigree  documents  for  compliance  and  evidence  of  risk.  Formal  risk  assessment  charts 
are  generated  for  each  identified  issue  and  the  scale  for  assessing  risk  is  shown  in  Figure 
2  below.  The  chart  includes  a  risk  matrix  with  probability  of  occurrence  on  the  x-axis  and 
mission  impact  on  the  y-axis.  The  IRRT  probability  ratings  run  from  Low  to  High.  Low 
probability  indicates  the  occurrence  is  highly  unlikely,  Medium  means  an  occurrence 
cannot  be  ruled  out,  and  High  indicates  an  occurrence  is  likely.  Each  of  the  ratings  is 
based  on  an  assessment  assuming  predicted  flight  environments  or  conditions. 
Intermediate  scores  such  as  Low-Medium  are  allowed  where  appropriate.  The  mission 
impact  categories  are  Negligible  meaning  no  mission  capability  degradation,  Moderate 
for  mission  capability  degradation,  and  Critical  for  loss  of  mission  or  safety  hazard.  The 
risk  ratings  are  Baseline,  Low,  Medium  and  High  with  intermediate  ratings  allowed  as 
noted  previously.  Baseline  risk  means  that  both  probability  and  impact  are  enveloped  by 
the  baseline  program  qualification.  Risk  is  no  higher  than  on  other  missions.  Low  means 
that  risk  is  higher  than  Baseline  but  is  bounded  by  supporting  test  and/or  analysis. 
Medium  refers  to  an  out-of- family  condition  or  out  of  test/flight  experience.  There  is 
reduced  confidence  in  supporting  tests  and/or  analysis  and  in  the  inherent  assumptions. 
High  represents  a  significant  out-of-family  condition  or  outside  of  test  and/or  flight 
experience.  Low  confidence  exists  in  supporting  test  and/or  analysis  and  the  inherent 
assumptions. 
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Figure  2.  The  IRRT  uses  this  chart  for  assessing  risk  for  issues  raised  during  our  reviews. 


A  program  risk  quad  chart  is  presented  to  the  SPO  undergoing  review  weekly.  It 
contains  information  such  as  the  review  plans,  schedule  and  the  items  in  work.  Risk 
reduction  recommendations  and  options  are  frequently  included.  This  chart  is  updated 
weekly  to  reflect  the  status  of  each  program.  The  IRRT  maintains  a  record  of  all  work 
performed  as  part  of  an  electronic  database.  This  assists  to  ensure  Lessons  Learned  and 
fleet  issues  are  not  forgotten.  The  Issue  Risk  Assessments  and  the  Program  Risk  Quad 
Chart  are  formally  briefed  as  part  of  the  flight  certification  process  to  the  SPO,  the 
Mission  Director,  the  Aerospace  President,  the  SMC  Chief  Engineer  and  the  SMC/CC  in 
various  pre-launch  reviews.  The  specific  reviews  supported  are  the  Mission  Readiness 
Review  (MRR),  the  Aerospace  President’s  Review  (APR),  and  the  Flight  Readiness 
Review  (FRR). 


V.  Examples  of  IRRT  Findings 

The  IRRT  has  performed  reviews  on  the  Evolved  Expendable  Launch  Vehicle 
(EELV)  Atlas  V  and  Delta  IV  boosters,  Heritage  boosters  such  as  Titan  II,  Delta  II  and 
Minotaur,  and  payloads  including  DSP,  GPS,  SBIRS,  DMSP,  Milstar,  AEHF,  WGS  and 
selected  space  test  programs.  There  have  been  numerous  findings  from  these  reviews  of 
which  the  following  are  samples. 

The  IRRT  Software  team  identified  severe  and  significant  shortcomings  in  flight 
software  development  and  execution  for  an  SMC  program.  It  provided  a  risk  reduction 
roadmap  that  the  SPO  adopted  and  has  used  to  track  risk  reduction.  In  addition,  the  IRRT 


UNCLASSIFIED 


6 


UNCLASSIFIED 


uncovered  grossly  inadequate  unit  level  tests  for  flight  software  and  recommended 
remedies  that  have  been  implemented  by  the  SPO  and  contractor. 

The  IRRT  diagnosed  the  root  cause  for  a  failure  in  a  phase  lock  loop  circuit  and 
recommended  corrective  action  to  the  SPO  and  contractor  that  were  accepted.  By 
examining  schematics,  the  IRRT  engineer  identified  the  location  of  an  open  circuit  that 
would  cause  the  behavior  observed  in  test.  He  visited  the  contractor’s  facility  where  the 
hardware  was  stored  and  examined  the  location  of  the  hardware  that  corresponded  to  the 
schematic  location.  He  found  that  the  behavior  would  come  and  go  by  pressing  on  the 
transistor  lead.  During  the  diagnosis  of  the  problem,  the  team  discovered  severe 
workmanship  problems  that  needed  corrective  action.  These  problems  were  corrected,  the 
payload  was  launched  and  continues  to  perform  as  expected. 

The  IRRT  Structural-Mechanical  panel  provided  recommendations  to  reduce  the 
risk  of  telescope  contamination  cover  latch  failure.  The  contamination  cover  protects  a 
telescope  aperture  and  is  a  single  point  failure.  The  SPO  and  contractor  accepted  design 
and  test  recommendations  from  the  IRRT  to  reduce  the  risk  of  mission  loss. 

During  booster  component  reviews,  the  IRRT  identified  two  motors  with  suspect 
nozzles.  The  first  motor  had  a  nozzle  that  was  dropped  off  a  pallet  then  subsequently 
installed  onto  a  motor.  The  IRRT  reviewed  the  motor  as  part  of  its  normal  review  process 
and  found  the  analysis  to  justify  the  use  of  the  dropped  nozzle  to  be  inadequate  since  it 
did  not  consider  dynamic  loads.  The  motor  manufacturer  and  the  SPO  both  agreed  that 
additional  work  was  required  to  clear  the  nozzle  prior  to  flight.  The  second  motor  had 
two  separate  sources  of  PAN  tape  used  to  manufacture  the  carbon  phenolic  exit  cone 
liner.  One  source  of  tape  was  used  on  half  of  the  liner  with  the  other  source  of  tape  used 
for  the  remainder.  The  two  sources  of  tape  had  substantially  different  mechanical 
properties  that  differed  by  as  much  as  a  factor  of  2.  The  motor  manufacturer  had  not  done 
any  analysis  to  determine  the  effects  of  the  exit  cone  liner  with  mechanical  properties  that 
changed  dramatically  midway  through  the  liner.  The  disposition  was  to  replace  the  motor 
and  set  it  aside  until  an  adequate  structural  analysis  of  the  exit  cone  liner  could  be 
performed. 

Another  issue  found  by  the  IRRT  was  a  lack  of  SRM  traceability  to  a  qualified 
baseline.  To  overcome  this  issue,  the  IRRT  reviewed  1 1  ship  sets,  including  several  that 
had  flown  to  evaluate  risk  of  flight.  The  resulting  review  showed  that  each  change  while 
not  necessarily  qualified  by  IRRT  standards,  was  reasonably  supported  by  flight  test, 
analysis  or  experience  from  other  programs.  The  launch  proceeded  with  an  elevated  risk 
rating  and  was  successful.  The  IRRT  has  recommended  the  contractor  show  proper 
traceability  to  the  qualification  baseline  or  perform  delta  qualifications  as  needed. 

VI.  Challenges  Forward 

For  an  IRRT  to  be  successful,  it  needs  a  senior  leadership  champion.  Six  years 
ago  when  the  team  was  founded,  the  IRRT  needed  the  help  of  the  center  commander  to 
gain  the  cooperation  of  the  SPOs  that  were  the  targets  of  the  IRRT’s  reviews.  Over  the 
ensuing  years,  the  IRRT  has  become  part  of  the  launch  process  and  an  accepted  member 
of  the  team.  As  the  failures  of  the  1990s  recede  into  the  past  and  current  budgets  receive 
greater  scrutiny,  there  is  a  danger  that  mission  success  may  lose  emphasis.  This  happened 
in  the  mid  1990s  under  Acquisition  Reform  with  disastrous  results.  As  noted  by  Lt. 
General  B.  Arnold  in  his  2004  congressional  hearing  testimony,  “...  I  want  to  reiterate 
the  Young  Panel  and  the  Space  Commission’s  findings  that  US  National  security  is 
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critically  dependent  on  space  capabilities  and  that  dependence  will  continue  to  grow.” 
Because  of  the  U.S.  dependence  on  communications  and  intelligence  gathering  satellites, 
the  OSS&E  emphasis  including  the  use  of  an  IRRT  must  not  be  marginalized  or 
eliminated. 


VII.  Summary  and  Conclusions 

The  SMC  IRRT  was  established  based  on  the  BAR  recommendations  and  the  Air 
Force’s  desire  to  refocus  on  OSS&E.  The  IRRT  has  conducted  technical  reviews  on  SMC 
launch  vehicles  and  spacecraft  since  2001.  SMC/CC  and  the  Board  of  Directors  approve 
the  funding  and  scope  for  the  IRRT.  The  IRRT  is  fully  established  and  uses  mature 
processes  that  have  been  demonstrated  to  be  effective.  The  IRRT  has  been  successful  in 
identifying  issues  and  suggesting  mitigations  to  lower  flight  risk.  The  IRRT  as  part  of 
Team  SMC  has  experienced  an  unprecedented  46  straight  mission  successes  in  a  row. 
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